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FIRST-ORDER  AND  SECOND-ORDER  NUMERICAL  METHODS 
FOR  OPTIMAL  CONTROL  PROBLEMS1 

A.  Miele 

Professor  of  Astronautics  and  Mathematical  Sciences 
Rice  University,  Houston,  Texas 

Abstract 

This  lecture  summarizes  recent  advances  in  the  area  of 
numerical  methods  for  optimal  control  problems,  with  particu¬ 
lar  emphasis  on  the  work  performed  by  the  Aero-Astronautics 
Group  of  Rice  University.  The  following  basic  problems  are 
considered:  problem  with  general  boundary  conditions,  prob¬ 
lem  with  nondifferential  constraints,  and  problem  with  mul¬ 
tiple  subarcs. 

First-order  algorithms  are  reviewed,  in  particular,  the 
sequential  ordinary  gradient-restoration  algorithm  and  thi 
sequential  conjugate  gradient-restoration  algorithm.  Second- 
order  algorithms  are  also  reviewed,  in  particular,  the  modi¬ 
fied  quasilinearization  algorithm.  Here,  the  optimal  initial 
choice  of  the  multipliers  is  discussed. 

Transformation  techniques  are  presented  by  means  of  which 
a  great  variety  of  problems  of  optimal  control  can  be  reduced 

1This  research  was  supported  by  the  Office  of  Scientific 
Research,  Office  of  Aerospace  Research,  United  States  Air 
Force,  Grant  No.  AF-AFOSR-76-3075 / and  by  the  National  Science 
Foundation,  Grant  No.  MCS-76-21657. 
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to  one  of  the  formulations  presented.  Specifically,  the  follow¬ 
ing  topics  are  treated:  time  normalization,  free  initial  state, 
problems  with  bounded  control,  problems  with  bounded  state,  and 
Chebyshev  minimax  problems. 

Key  Words 

Optimal  control,  numerical  analysis,  numerical  methods, 
computing  methods,  computing  techniques,  sequential  ordinary 
gradient-restoration  algorithm,  sequential  conjugate  gradient- 
restoration  algorithm,  modified  quasilinearization  algorithm, 
problems  with  bounded  control,  problems  with  bounded  state, 
Chebyshev  minimax  problems. 

Introduction 

In  every  branch  of  science  and  engineering,  there  exist 
systems  which  are  controllable,  that  is, they  can  be  made  to 
behave  in  different  ways  depending  on  the  will  of  the  opera¬ 
tor.  Every  time  the  operator  of  a  system  exerts  an  option 
(a  choice  in  the  distribution  of  the  controls  governing  the 
system) ,  he  produces  a  change  in  the  distribution  of  the  states 
occupied  by  the  system  and,  hence,  a  change  in  the  final  state. 
Therefore,  it  is  natural  to  pose  the  following  question:  Among 
all  the  admissible  options,  what  is  the  particular  option  which 
renders  the  system  optimum?  As  an  example,  what  is  the  option 
which  minimizes  the  difference  between  the  final  value  and  the 
initial  value  of  an  arbitrarily  specified  function  of  the  state 


MDP-79-N6-03 


of  the  sy e.tam?  The  body  of  knowledge  covering  problems  of 
this  type  is  called  optimal  control  theory.  Applications 
occur  in  every  field  of  science  and  engineering,  and  also 
economics. 

It  must  be  noted  that  only  a  minority  of  current  technical 
problems  can  be  solved  by  purely  analytical  methods.  Hence,  it 
is  important  to  develop  numerical  techniques  enabling  one  to 
solve  optimal  control  problems  on  a  digital  computer.  These 
numerical  techniques  can  be  classified  into  two  groups:  first- 
order  methods  and  second-order  methods.  First-order  methods 
(or  gradient  methods)  are  those  techniques  which  employ  at  most 
the  first  derivatives  of  the  functions  under  consideration. 
Second-order  methods  (or  quasilinearization  methods)  are  those 
techniques  which  employ  at  most  the  second  derivatives  of  the 
functions  under  consideration. 

Both  gradient  methods  and  quasilinearization  methods  re¬ 
quire  the  solution  of  a  linear,  two-point  boundary-value  problem 
(and  sometimes  the  solution  of  a  linear,  multi-point  boundary- 
value  problem)  at  every  iteration.  This  being  the  case,  prog¬ 
ress  in  the  area  of  numerical  methods  for  differential  equa¬ 
tions  is  essential  to  the  efficient  solution  of  optimal  control 
problems  on  a  digital  computer.  _ 

rom — i 
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Problems 

For  the  sake  of  discussion,  assume  that  the  system  under 
consideration  is  described  by  the  state  x(t),  the  control 
u(t),  and  the  parameter  it.  Also,  assume  that  the  interval  of 
integration  is  fixed,  the  initial  point  is  given,  the  state  is 
unbounded,  the  control  is  unbounded,  and  the  parameter  is  un¬ 
bounded.  Under  these  conditions,  we  consider  three  basic  prob¬ 
lems,  called  Problem  Pi,  Problem  P2,  and  Problem  P3  for  easy 
identification. 


Problem  PI.  Minimize  the  functional 

I  «  f  (x,u,~,t)dt+g(x(l)  ,-n)  ,  (1) 

subject  to  the  constraints 

X  =  0  (X,U,TT,t)  ,  0<t<l,  (2) 

x (0)  =  given,  (3) 

0  (X  (1)  ,  7T )  “  0.  (4) 


The  minimization  of  the  functional  (1)  must  be  performed  with 
respect  to  the  state  x(t),  the  control  u(t),  and  the  parameter 
tt ,  with  the  understanding  that  the  time  t  is  a  scalar  and  x,u,7r 
are  vectors  of  appropriate  dimensions.  The  functions  f,g 
appearing  in  the  functional  (1)  are  scalar,  and  the  functions 
0,0  appearing  in  the  constraints  (2) -(4)  are  vectors  of  appro¬ 
priate  dimensions. 
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Problem  P2.  Minimize  the  functional 

1“  f  (x,u,ir,t)dt  +  g(x(l)  ,v) ,  (5) 

subject  to  the  constraints 


X  =  d  (x,U,TT,  t)  ,  0  <  t  <  1, 

(6) 

S  (x,u, 7r , t)  =  0,  0  <  t  <  1, 

(7) 

x (0)  =  given. 

(3) 

^(x(l)  ,  7T )  =  0  . 

(9) 

Clearly,  Problem  P2  differs  from  Problem  Pi  because  of  the 
presence  of  the  nondifferential  constraint  (7) ,  to  be  satisfied 
everywhere  along  the  interval  of  integration.  The  function  S 
appearing  in  Eq.  (7)  is  a  vector  of  appropriate  dimension. 

Problem  P3.  This  is  a  modification  of  Problem  P2  which 
arises  whenever  the  trajectories  of  the  system  include  several 
subarcs,  with  the  following  provision:  the  analytical  form  of 
the  functions  f,<J),S  might  change  from  one  subarc  to  another. 

Algorithms 

Over  the  past  decade,  the  Aero-Astronautics  Group  of  Rice 
University  has  successfully  developed  several  algorithms  for 
solving  Problems  Pi,  P2,  P 3  on  a  digital  computer.  Notable 
among  these  algorithms  are  the  sequential  ordinary  gradient- 
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restoration  algorithm  (SOGRA,  Refs.  1-4) ,  the  sequential 
conjugate  gradient-restoration  algorithm  (SCGRA,  Refs.  5-3), 
and  the  modified-quasilinearizatior.  algorithm  (MQA,  Refs.  S-12). 
Both  SOGRA  and  SCGRA  are  first-order  algorithms  which  have  a 
cyclical  construction.  Each  cycle  includes  a  gradient  phase 
(one  iteration)  and  a  restoration  phase  (several  iterations) 
and  is  constructed  in  such  a  way  that  the  decrease  of  the  fun¬ 
ctional  I  is  guaranteed,  while  the  constraints  are  satisfied 
to  a  predetermined  tolerance.  On  the  other  hand,  MQA  is  a 
second-order  algorithm  which  has  an  iterative  construction. 

Each  iteration  is  constructed  in  such  a  way  that  the  decrease 
of  the  total  error  in  the  system  (the  sum  of  the  error  in  the 
feasibility  equations  and  the  error  in  the  optimality  condi¬ 
tions)  is  guaranteed. 

Each  iteration  of  SOGRA,  SCGRA,  and  MQA  requires  the  sol¬ 
ution  of  a  linear  two-point  boundary-value  problem  or  a  linear 
multi-point  boundary-value  problem.  In  this  connection,  seve¬ 
ral  techniques  can  be  useful  (method  of  adjoint  variables, 
method  of  complementary  functions,  method  of  particular  solu¬ 
tions,  and  Riccati  transformation).  Among  these,  the  method 
of  particular  solutions  (Refs.  13-20)  has  been  employed  in 
the  solution  of  the  LTP-BVP  or  LMP-BVP  associated  with  SOGRA, 
SCGRA,  and  MQA. 


Transformation  Techniques 

In  spite  of  the  generality  of  the  previous  work,  a  wide 
variety  of  problems  of  optimal  control  exist  which  are  not 
covered  by  the  above  standard  schemes.  Whenever  a  nonstandard 
situation  arises,  one  has  two  choices:  (i)  to  develop  an  ad-hoc 
algorithm  capable  of  solving  the  nonstandard  problem  on  a  digi¬ 
tal  computer;  or  (ii)  to  develop  transformation  techniques  such 
that  the  nonstandard  problem  is  brought  within  the  frames  of 
Problems  PI,  P2,P3.  This  can  be  achieved  through  proper 

augmentation  and  redefinition  of  the  state  vector  x(t),  the 
control  vector  u(t),  and  the  parameter  vector  vr,  as  well  as 
through  proper  augmentation  and  redefinition  of  the  constrain¬ 
ing  relations  (see,  for  example.  Refs.  21-24). 

By  means  of  transformation  techniques,  the  following  prob¬ 
lems  of  optimal  control  can  be  brought  within  the  frames  of 
Problems  Pi,  P2,  P3:  (i)  problems  with  variable  final  time; 

(ii)  problems  with  arbitrary  conditions  on  the  initial  state; 

(iii)  problems  with  control  equality  constraints;  (iv)  problems 
with  state  equality  constraints;  (v)  problems  with  state- 
derivative  equality  constraints;  (vi)  problems  with  control 
inequality  constraints;  (vii)  problems  with  state  inequality 
constraints;  (viii)  problems  with  state-derivative  inequality 
constraints;  and  (ix)  minimax  problems  of  optimal  control. 
Therefore,  by  transformation  techniques,  the  scope  and  range 

of  SOGRA,  SCGRA,  and  MQA  can  be  increased  dramatically. 
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